{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "## 4.3 利用数组进行数据处理 Data Processing Using Arrays\r\n",
    "\r\n",
    "+ NumPy数组可以将数据处理任务表述为简洁的数组表达式，而不需要编写循环\r\n",
    "+ 利用数组表达式代替循环的做法称之为：向量化（Vectorization）\r\n",
    "+ 一般来说，向量化的数组操作比等价的纯Python循环操作要快很多（一两个数量级）"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "# 模块导入\r\n",
    "import os, sys\r\n",
    "sys.path.append(os.path.dirname(os.getcwd()))\r\n",
    "import numpy\r\n",
    "import matplotlib.pyplot\r\n",
    "from dependency import arr_info"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 绪  论"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# 生成二维网格坐标的方法\r\n",
    "\r\n",
    "points = numpy.arange(-5, 6)\r\n",
    "\r\n",
    "points_x, points_y = numpy.meshgrid(points, points)\r\n",
    "\r\n",
    "arr_info([ points, points_x, points_y ])\r\n",
    "\r\n",
    "for i, j in zip(range(-5, 6), range(-5, 6)):\r\n",
    "    print(points_x[i,j], points_y[i,j])"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# 计算一组网格型数据的函数值\r\n",
    "\r\n",
    "points = numpy.arange(-5, 5, 0.01)      # 生成1000个间隔为0.01的点\r\n",
    "x, y = numpy.meshgrid(points, points)   # 由一维的points生成二维坐标数组\r\n",
    "\r\n",
    "z = numpy.sqrt(x**2 + y**2)     # 计算平方和开方的值\r\n",
    "\r\n",
    "# matplotlib.pyplot.scatter(x, y, c=z, cmap=matplotlib.cm.Blues)  # 耗时：20.6s\r\n",
    "matplotlib.pyplot.imshow(z, cmap=matplotlib.cm.Blues)   # 耗时：0.3s；灰度图：cmap=matplotlib.cm.gray\r\n",
    "matplotlib.pyplot.colorbar()   # 图例\r\n",
    "matplotlib.pyplot.title(\"Image plot of $\\sqrt{x^2 + y^2}$ for a grid of values\")  # 图线标题"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 4.3.1 将条件逻辑表述为数组运算 Expressing Conditional Logic as Array Operations\r\n",
    "\r\n",
    "`numpy.where(condition, a, b)`函数是三元表达式 `x if(condition) else y` 的矢量化版本。其中，`a`、`b` 可以为数组，也可以为标量。\r\n",
    "\r\n",
    "该函数相比逻辑表达式有以下优点：\r\n",
    "\r\n",
    "+ 对大数据组的处理速度比纯Python语法更快\r\n",
    "+ 可以处理多维数组，上述三元表达式无法对多维数组进行处理"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# 三元表达式：x if(condition) else y\r\n",
    "\r\n",
    "ary1_x = numpy.array([1.1, 1.2, 1.3, 1.4, 1.5])\r\n",
    "ary1_y = numpy.array([2.1, 2.2, 2.3, 2.4, 2.5])\r\n",
    "condition = numpy.array([True, False, True, True, False])\r\n",
    "\r\n",
    "result_1 = [(x if(cond) else y) for x, y, cond in zip(ary1_x, ary1_y, condition)]     # 列表推导式\r\n",
    "arr_info([result_1])    # 返回值为 list 对象，而非 ndarray"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# numpy.where() 函数\r\n",
    "\r\n",
    "result_2 = numpy.where(condition, ary1_x, ary1_y)\r\n",
    "\r\n",
    "arr_info([result_2])   # 返回值为 ndarray"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# 实用例子：所有正值替换为2，所有负值替换为-2\r\n",
    "\r\n",
    "arr1_2 = numpy.random.randn(3, 4)\r\n",
    "result_3 = numpy.where(arr1_2 > 0, \"Positive\", \"Negative\")\r\n",
    "\r\n",
    "arr_info( [arr1_2, result_3] )"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# 更复杂的例子\r\n",
    "\r\n",
    "result_4 = []\r\n",
    "cond_1 = numpy.array([True, False, True, True, False])\r\n",
    "cond_2 = numpy.array([False, False, True, False, False])\r\n",
    "\r\n",
    "# 利用纯Python来编写\r\n",
    "for i in range(5):\r\n",
    "    if (cond_1[i] & cond_2[i]):\r\n",
    "        result_4.append(0)\r\n",
    "    elif (cond_1[i]):\r\n",
    "        result_4.append(1)\r\n",
    "    elif (cond_2[i]):\r\n",
    "        result_4.append(2)\r\n",
    "    else:\r\n",
    "        result_4.append(3)\r\n",
    "\r\n",
    "# 利用 numpy.where() 函数编写（多次嵌套）\r\n",
    "result_5 = numpy.where(  (cond_1 & cond_2), 0, numpy.where( cond_1, 1, numpy.where(cond_2, 2, 3) )  )\r\n",
    "\r\n",
    "# 布尔值在计算过程中可以被当作 0 或 1 处理（不推荐，可读性太差）\r\n",
    "result_6 = 1 * (cond_1 & ~cond_2) + 2 * (~cond_1 & cond_2) + 3 * ~(cond_1 | cond_2)\r\n",
    "\r\n",
    "arr_info([result_4, result_5, result_6])"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 4.3.2 数学方法与统计方法 Mathematical and Statistical Methods\r\n",
    "\r\n",
    "可以对整个数组或某个轴向上的数据进行统计计算\r\n",
    "\r\n",
    "聚合计算（Aggregation）：通常也称为约简（Reduction），例如 `sum`、`mean`、`std` 等\r\n",
    "\r\n",
    "数组统计的类方法（只能针对ndarray对象）：\r\n",
    "\r\n",
    "+ `array.sum()`：对数组元素或某个轴向的数据求和，零长度数组求和为 0；\r\n",
    "+ `array.mean()`：算数平均值，零长度数组则为 NaN；\r\n",
    "+ `array.std()`：标准差，自由度默认为 n，即 `ddof=0`；\r\n",
    "+ `array.var()`：方差，自由度默认为 n，即 `ddof=0`；\r\n",
    "+ `array.max()`，`array.min()`：最大值，最小值\r\n",
    "+ `array.argmax()`，`array.argmin()`：最大值元素的索引，最小值元素的索引；\r\n",
    "+ `array.cumsum()`：所有元素的累计和\r\n",
    "+ `array.cumprod()`：所有元素的累计积\r\n",
    "\r\n",
    "可以调用函数实现相同的功能（可以对其他可迭代对象进行统计）：\r\n",
    "\r\n",
    "+ `numpy.sum(array)`：对数组元素或某个轴向的数据求和，零长度数组求和为 0；\r\n",
    "+ `numpy.mean(array)`：算数平均值，零长度数组则为 NaN；\r\n",
    "+ `numpy.std(array)`：标准差，自由度默认为 n；\r\n",
    "+ `numpy.var(array)`：方差，自由度默认为 n；\r\n",
    "+ `numpy.max(array)`，`array.min(array)`：最大值，最小值\r\n",
    "+ `numpy.argmax(array)`，`array.argmin(array)`：最大值元素的索引，最小值元素的索引；\r\n",
    "+ `numpy.cumsum(array)`：所有元素的累计和\r\n",
    "+ `numpy.cumprod(array)`：所有元素的累计积\r\n",
    "\r\n",
    "没有对应的类方法：\r\n",
    "+ `numpy.cov(array)`：计算协方差矩阵\r\n",
    "+ `numpy.corrcoef(array)`：计算相关系数矩阵\r\n",
    "\r\n",
    "【注意】\r\n",
    "\r\n",
    "+ 默认计算整个数组的统计信息，可以指定按哪一个轴计算，例如以 `axis=0` 的方向计算，即得到的是每列的统计值，\r\n",
    "+ `.cumsum()`、`.cumprod()` 类方法（或相应的Numpy函数）将产生一个由中间结果组成的数组\r\n",
    "+ `.std()`、`.var()` 类方法（或相应的Numpy函数）默认计算总体标准差、总体方差（即自由度为n），如果要计算修正样本标准差、修正样本方差，需要指定参数 `ddof=1`（即自由度为n-1），不指定则默认 `ddof=0`；"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# 统计方法\r\n",
    "\r\n",
    "# ary2_1 = numpy.random.randn(4, 5)   # 生成符合正态分布的随机数\r\n",
    "ary2_1 = numpy.array([numpy.arange(-5, 6), numpy.arange(-10, 1), numpy.arange(0, 11)])\r\n",
    "\r\n",
    "arr_info([ary2_1])\r\n",
    "\r\n",
    "# 两种等价的写法\r\n",
    "print(\"平均值:\", numpy.mean(ary2_1, axis=1))    # 调用NumPy函数\r\n",
    "print(\"平均值:\", ary2_1.mean(axis=1))           # 调用ndarray的类方法\r\n",
    "print(\"\")\r\n",
    "\r\n",
    "# 标准差与方差指定自由度\r\n",
    "print(\"总体方差:\", ary2_1.var())\r\n",
    "print(\"修正样本方差:\", ary2_1.var(ddof=1))\r\n",
    "print(\"(n-1)/n * Sn^* =\", (32/33) * ary2_1.var(ddof=1))\r\n",
    "print(\"\")\r\n",
    "# Numpy默认是除以样本数，求的是母体标准差；Matlab默认是除以(样本-1)，求的是样本标准差。ddof相当于其中减去的数字1参数，默认为0。\r\n",
    "\r\n",
    "# 以整个array计算\r\n",
    "print(\"【其他统计信息】整个array\")\r\n",
    "print(\"求  和:\", numpy.sum(ary2_1))\r\n",
    "print(\"最大值:\", ary2_1.max())\r\n",
    "print(\"最小值:\", ary2_1.min())\r\n",
    "print(\"最大值索引:\", ary2_1.argmax())\r\n",
    "print(\"最小值索引:\", ary2_1.argmin())\r\n",
    "print(\"累计和:\")\r\n",
    "arr_info([ ary2_1.cumsum() ])\r\n",
    "print(\"累计积:\")\r\n",
    "arr_info([ ary2_1.cumprod() ])"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# axis=0 轴向\r\n",
    "\r\n",
    "arr_info([ary2_1])\r\n",
    "print(\"【统计信息】axis=0 轴向\")\r\n",
    "print(\"求  和:\", ary2_1.sum(0))\r\n",
    "print(\"平均值:\", ary2_1.mean(0))\r\n",
    "print(\"总体标准差:\", ary2_1.std(0))\r\n",
    "print(\"修正样本标准差:\", ary2_1.std(axis=0, ddof=1))\r\n",
    "print(\"总体方差:\", ary2_1.var(0))\r\n",
    "print(\"修正样本方差:\", ary2_1.var(axis=0, ddof=1))\r\n",
    "print(\"最大值:\", ary2_1.max(0))\r\n",
    "print(\"最小值:\", ary2_1.min(0))\r\n",
    "print(\"最大值索引:\", ary2_1.argmax(0))\r\n",
    "print(\"最小值索引:\", ary2_1.argmin(0))\r\n",
    "print(\"累计和:\")\r\n",
    "arr_info([ ary2_1.cumsum(0) ])\r\n",
    "print(\"累计积:\")\r\n",
    "arr_info([ ary2_1.cumprod(0) ])"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# axis=1轴向\r\n",
    "\r\n",
    "arr_info([ary2_1])\r\n",
    "print(\"【统计信息】axis=1 轴向\")\r\n",
    "print(\"求  和:\", ary2_1.sum(1))\r\n",
    "print(\"平均值:\", ary2_1.mean(1))\r\n",
    "print(\"总体标准差:\", ary2_1.std(1))\r\n",
    "print(\"修正样本标准差:\", ary2_1.std(axis=1, ddof=1))\r\n",
    "print(\"总体方差:\", ary2_1.var(1))\r\n",
    "print(\"修正样本方差:\", ary2_1.var(axis=1, ddof=1))\r\n",
    "print(\"最大值:\", ary2_1.max(1))\r\n",
    "print(\"最小值:\", ary2_1.min(1))\r\n",
    "print(\"最大值索引:\", ary2_1.argmax(1))\r\n",
    "print(\"最小值索引:\", ary2_1.argmin(1))\r\n",
    "print(\"累计和:\")\r\n",
    "arr_info([ ary2_1.cumsum(1) ])\r\n",
    "print(\"累计积:\")\r\n",
    "arr_info([ ary2_1.cumprod(1) ])"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 4.3.3 用于布尔数组的方法 Methods for Boolean Arrays\r\n",
    "\r\n",
    "在上述统计方法中，布尔值会被强制转换为 1（`True`）和 0（`False`）。\r\n",
    "\r\n",
    "`.sum()` 常用来统计布尔数组中 `True` 的个数\r\n",
    "\r\n",
    "对布尔数组非常有用的两个方法（或对应的函数），但这两个函数也可以用于非布尔数组，非零元素将被视为 `True`：\r\n",
    "\r\n",
    "+ `array.any()`：检测数组中是否存在任何 `True`；\r\n",
    "+ `array.all()`：检测数组中是否均为 `True`；"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# 统计 True 的个数\r\n",
    "\r\n",
    "ary3_1 = numpy.random.randn(20)\r\n",
    "\r\n",
    "positive_count = (ary3_1 > 0).sum()\r\n",
    "\r\n",
    "print(\"正数的个数:\", positive_count)\r\n",
    "arr_info([ary3_1])"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# any() all()\r\n",
    "\r\n",
    "ary3_1 = numpy.array([True, False, True, True, False])\r\n",
    "\r\n",
    "print(\"数组中是否存在任何“True”:\", ary3_1.any())\r\n",
    "print(\"数组中是否均为“Ture”:\", ary3_1.all())"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 4.3.4 排  序 Sorting\r\n",
    "\r\n",
    "+ `array.sort()`：类方法，无返回值\r\n",
    "+ `numpy.sort(array)`：函数，返回排序后数组的副本"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# 排序\r\n",
    "\r\n",
    "# ary4_1 = numpy.random.randn(10)\r\n",
    "ary4_1 = numpy.array([3, 6, 4, 9, 5, 2, 0, 8, 7, 1])\r\n",
    "\r\n",
    "arr_info([ary4_1])\r\n",
    "ary4_1.sort()   # 如果直接调用 print(ary4_1.sort()) 会显示None，这是因为.sort()方法没有返回值！\r\n",
    "arr_info([ary4_1, numpy.sort(ary4_1)])"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# 实例：分位数\r\n",
    "\r\n",
    "ary4_2 = numpy.random.randn(100)\r\n",
    "\r\n",
    "arr_info([ary4_2])\r\n",
    "ary4_2.sort()\r\n",
    "arr_info([ary4_2])\r\n",
    "\r\n",
    "quantile_5 = ary4_2[ int(0.05 * len(ary4_2)) ]  # 5%分位数\r\n",
    "print(\"5% 分位数:\", quantile_5)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 4.3.5 唯一化以及其他集合逻辑 Unique and Other Set Logic\r\n",
    "\r\n",
    "+ `numpy.unique(array)`：唯一化数组，并返回排序后的结果；\r\n",
    "+ `numpy.intersect1d(array_1, array_2)`：计算两个数组中的公共元素，并返回排序后的结果；\r\n",
    "+ `numpy.union1d(array_1, array_2)`：计算两个数组中的并集，并返回排序后的结果；\r\n",
    "+ `numpy.in1d(array_1, array_2)`：得到一个表示 “第一个数组的元素是否包含于第二个数组” 的布尔数组；\r\n",
    "+ `numpy.setdiff1d(array_1, array_2)`：集合的差，即第一个数组的元素且不包含第二个数组的元素；\r\n",
    "+ `numpy.setxor1d(array_1, array_2)`：对称差（异或），存在于一个数组中，但不同时存在于两个数组中的元素；"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# 唯一化\r\n",
    "\r\n",
    "names = numpy.array([\"Kiana\", \"Kiana\", \"Kaslana\", \"Kiana Kaslana\", \"Kiana\", \"Raiden Mei\", \"Bronya Zaychik\" ])\r\n",
    "\r\n",
    "name_unique = numpy.unique(names)   # Numpy唯一化并排序的函数\r\n",
    "names_sorted = sorted(set(names))   # 等价的纯Python写法\r\n",
    "\r\n",
    "arr_info([ names, name_unique, names_sorted ])"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "# intersect1d()函数，in1d()函数\r\n",
    "\r\n",
    "ary5_1 = numpy.array([6, 0, 0, 3, 2, 5, 6])\r\n",
    "\r\n",
    "ary5_2 = numpy.intersect1d( ary5_1, [2, 3, 6, 7, 9] )\r\n",
    "ary5_3 = numpy.in1d( ary5_1, [2, 3, 6, 7, 9] )\r\n",
    "\r\n",
    "arr_info([ary5_1, ary5_2, ary5_3])"
   ],
   "outputs": [],
   "metadata": {}
  }
 ],
 "metadata": {
  "orig_nbformat": 4,
  "language_info": {
   "name": "python",
   "version": "3.9.6",
   "mimetype": "text/x-python",
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "pygments_lexer": "ipython3",
   "nbconvert_exporter": "python",
   "file_extension": ".py"
  },
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.9.6 64-bit ('DataAnalysis': venv)"
  },
  "interpreter": {
   "hash": "0fe90ac710750590bd916650f441b3b2233faa2658175d6302b22e90f225e38d"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}